Image processing method and apparatus, and image display method and apparatus, with variable interpolation spacing

ABSTRACT

An image is processed by detecting pixel-to-pixel variations in brightness level, generating high spatial frequency information related to the variations, setting interpolation points with a spacing that varies according to the high spatial frequency information, and generating new pixels by interpolation at the interpolation points. By increasing the zoom ratio in one part and reducing the zoom in another part of each edge in a continuous manner, this method can mitigate edge degradation when an image is enlarged or reduced, without introducing discontinuities or other image artifacts. It also provides a convenient way to adjust edge sharpness in an image.

BACKGROUND OF THE INVENTION

This invention relates to an image processing method and apparatus andan image display method and apparatus, more particularly to methods andapparatus that are useful for expanding and reducing digitized images,and for controlling edge sharpness.

Digital image expansion and reduction are processes that change thenumber of picture elements (pixels) in an image. These processes areoften necessary. For example, an image generated in the widely used640-by-480-pixel format may need to be expanded for display on a1024-by-768-pixel liquid crystal screen, or reduced for display in awindow occupying only part of that screen.

FIGS. 1A and 1B illustrate the conventional expansion of an image by afactor of three. The horizontal axis in these drawings represents ahorizontal row of pixels in the image; the vertical axis represents thepixel data values, indicating pixel brightness levels. Before expansion,the row of pixels has segments (h) of uniform brightness, separated byedges (j, k) at which the brightness level changes, as shown in FIG. 1A.The conventional expansion process expands all of these segments andedges identically by a factor of three, so that the edges (j1, k1) inthe expanded image, shown in FIG. 1B, are not as sharp as the edges inthe original image.

The pixel values in the expanded picture are determined byinterpolation, which is performed by a spatial filtering operation asillustrated in FIG. 2. The horizontal axis again represents horizontalposition; the vertical axis now represents the value of an interpolationfilter response characteristic F(x). If p(n) and p(n+1) are twoconsecutive pixels in the input image, q(n) is a pixel disposed at aposition between them in the output image, the distance from p(n) top(n+1) is equal to unity, and the distance from p(n) to q(m) is equal tor, then the brightness level of pixel q(m) is calculated as follows:

q(m)=(F(r)×p(n))+(F(1−r)×p(n+1))

FIG. 3 illustrates schematically how this filtering calculationgenerates seven output pixels (q1 to q7) from three input pixels (p1 top3).

The filter response characteristic need not be linear. An interpolationfilter with a nonlinear characteristic, as illustrated in FIG. 4, issometimes used to enhance the sharpness of edges (e.g., j1 and k1) inthe output image. This type of edge enhancement, however, leads tofurther problems such as undershoot (pre-shoot) and overshoot.

Japanese Unexamined Patent Application No. 9-266531 discloses a schemethat provides several filter response characteristics, as illustrated inFIGS. 5A, 5B, and 5C, and switches among them according to the type ofimage area being processed. When this filter-switching scheme is used,however, there are problems of discontinuities at the points at whichswitching takes place.

Problems also occur when an image is reduced by conventional methods.The quality of edges is degraded because of pixel dropout.

SUMMARY OF THE INVENTION

One object of this invention is to mitigate edge degradation when animage is expanded or reduced.

Another object is to control edge sharpness in an image.

The invention provides a method of processing an image, formed frominput pixels, by the following steps:

(a) detecting pixel-to-pixel variations in the brightness levels of theinput pixels in at least one direction in the image, thereby generatinghigh spatial frequency information;

(b) setting interpolation points with a spacing that varies according tothe high spatial frequency information; and

(c) generating output pixels from the input pixels by interpolation atthe interpolation points.

Step (b) preferably assigns a basic value to the spacing of theinterpolation points in parts of the image in which the brightness levelis uniform. As for portions of the image in which the brightness levelvaries, step (b) may divide each such portion into a first part and asecond part, reduce the interpolation spacing in the first part, andincrease the interpolation spacing in the second part. Alternatively,step (b) may divide each such portion into a first part, a second part,and a third part, reduce the interpolation spacing in the first andthird parts, and increase the interpolation spacing in the second part.

Step (a) includes, for example, calculating a first derivative of thebrightness levels in the above-mentioned one direction. Step (a) mayalso include calculating a second or third derivative of the brightnesslevels, or performing a filtering operation to obtain a certain spatialfrequency component of the image, by performing two low-pass filteringoperations with different cut-off frequencies and taking the differencebetween the resulting two low spatial frequency components, for example.

Alternatively, step (a) may include detecting patterns of variation inthe brightness levels of the input pixels. The detected patternsdescribe, for example, the polarity of pixel-to-pixel changes inbrightness level, or the polarity and magnitude of the changes. Thechanges thus described may be, for example, changes in the brightnesslevels of three consecutive pixels, or changes in the brightness levelsof five consecutive pixels.

The invention also provides a machine-readable medium storing amachine-executable program for processing an image by the inventedmethod.

The invention moreover provides an image-processing apparatus forprocessing an image formed from input pixels. The apparatus includes afirst processing unit that detects pixel-to-pixel variations in thebrightness levels of the input pixels in at least one direction in theimage, and generates high spatial frequency information; a secondprocessing unit that sets interpolation points with a spacing varyingaccording to the high spatial frequency information; and a thirdprocessing unit that generates output pixels from the input pixels byinterpolation at the interpolation points.

The second processing unit preferably assigns a basic value to theinterpolation spacing in parts of the image in which the brightnesslevel is uniform. In processing portions of the image in which thebrightness level changes, the second processing unit may divide eachsuch portion into a first part and a second part, reduce theinterpolation spacing in the first part, and increase the interpolationspacing in the second part. Alternatively, it may divide each suchportion into a first part, a second part, and a third part, reduce theinterpolation spacing in the first and third parts, and increase theinterpolation spacing in the second part.

As high spatial frequency information, the first processing unit maycalculate a first derivative of the brightness levels in theabove-mentioned one direction. The first processing unit may alsocalculate a second or third derivative of the brightness levels, orperform a filtering operation to obtain a certain spatial frequencycomponent of the image, by performing two low-pass filtering operationswith different cut-off frequencies and taking the difference between theresulting two low spatial frequency components, for example.

Alternatively, the first processing unit may detect patterns ofvariation in the brightness levels of the input pixels. The detectedpatterns describe, for example, the polarity of pixel-to-pixel changesin brightness level, or the polarity and magnitude of the changes. Thechanges thus described may be, for example, changes in the brightnesslevels of three consecutive pixels, or changes in the brightness levelsof five consecutive pixels.

The invention furthermore provides an image display apparatus fordisplaying an image formed from input pixels. The image displayapparatus includes a memory unit that stores the brightness levels ofthe input pixels; a first processing unit that detects pixel-to-pixelvariations in the brightness levels in at least one direction in theimage, thereby generating high spatial frequency information; a secondprocessing unit that sets interpolation points with a spacing thatvaries according to the high spatial frequency information; a thirdprocessing unit that generates output pixels from the input pixels byinterpolation at the interpolation points; and a display unit thatdisplays the output pixels.

The second processing unit preferably assigns a basic value to theinterpolation spacing in parts of the image in which the brightnesslevel is uniform. In processing portions of the image in which thebrightness level changes, the second processing unit may divide eachsuch portion into a first part and a second part, reduce theinterpolation spacing in the first part, and increase the interpolationspacing in the second part. Alternatively, it may divide each suchportion into a first part, a second part, and a third part, reduce theinterpolation spacing in the first and third parts, and increase theinterpolation spacing in the second part.

As high spatial frequency information, the first processing unit maycalculate a first derivative of the brightness levels in theabove-mentioned one direction. The first processing unit may alsocalculate a second or third derivative of the brightness levels, orperform a filtering operation to obtain a certain spatial frequencycomponent of the image, by performing two low-pass filtering operationswith different cut-off frequencies and taking the difference between theresulting two low spatial frequency components, for example.

Alternatively, the first processing unit may detect patterns ofvariation in the brightness levels of the input pixels. The detectedpatterns describe, for example, the polarity of pixel-to-pixel changesin brightness level, or the polarity and magnitude of the changes. Thechanges thus described may be, for example, changes in the brightnesslevels of three consecutive pixels, or changes in the brightness levelsof five consecutive pixels.

Varying the spacing of the interpolation points provides a way tocontrol edge sharpness without introducing discontinuities or otherunwanted image artifacts.

Increasing the interpolation spacing mitigates the loss of edgesharpness that occurs when an image is expanded.

Decreasing the interpolation spacing mitigates the drop-out problem thatoccurs when an image is reduced.

Dividing an image portion into multiple parts and increasing theinterpolation spacing in at least one part while decreasing theinterpolation spacing in another part enables the loss-of-sharpnessproblem to be mitigated in edge expansion, and the drop-out problem tobe mitigated in image reduction, without changing the overall expansionor reduction ratio.

Use of the first, second, and third derivatives as high spatialfrequency information enables continuously-varying information to beobtained by simple arithmetic operations. Filtering also enables suchinformation to be obtained by relatively simple arithmetic operations.

Detecting patterns of variation in the brightness levels of the inputpixels enables high-frequency information to be generated by simplearithmetic and logic operations. Detecting patterns in the polarity ofchanges over three consecutive pixels provides adequate continuity, butthe continuity can be improved by also detecting the magnitude of thechanges, or by detecting patterns over five consecutive pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

In the attached drawings:

FIG. 1A illustrates pixel data values in part of an image;

FIG. 1B illustrates corresponding pixel data values in a conventionallyexpanded image;

FIG. 2 illustrates an interpolation filter response characteristic;

FIG. 3 illustrates the generation of expanded image data by conventionalinterpolation with the characteristic in FIG. 2;

FIG. 4 illustrates a conventional non-linear interpolation filterresponse characteristic;

FIGS. 5A, 5B, and 5C illustrate a selection of conventionalinterpolation filter response characteristics;

FIG. 6 illustrates an image display apparatus embodying the invention;

FIG. 7A illustrates pixel data values in part of an image, with edgesdivided into leading and trailing parts;

FIG. 7B illustrates corresponding pixel data values after imageexpansion according to several embodiments of the invention;

FIG. 8A illustrates pixel data values in part of an image, with edgesdivided into leading and trailing parts;

FIG. 8B illustrates corresponding pixel data values after imagereduction according to several embodiments of the invention;

FIG. 9 illustrates the internal structure of the zoom processor in FIG.6 according to a first embodiment of the invention;

FIGS. 10A, 10B, 10C, 10D, and 10E are spatial waveform diagramsillustrating the operation of the first embodiment;

FIG. 11 illustrates interpolation in the first embodiment at athree-pixel edge;

FIG. 12 illustrates the internal structure of the zoom processor in avariation of the first embodiment;

FIG. 13 illustrates the internal structure of the zoom processor in asecond embodiment of the invention;

FIGS. 14A, 14B, 14C, and 14D are spatial waveform diagrams illustratingthe operation of the second embodiment;

FIG. 15 illustrates interpolation in the second embodiment at atwo-pixel edge;

FIG. 16 illustrates the internal structure of the zoom processor in avariation of the second embodiment;

FIG. 17 illustrates the internal structure of the zoom processor in athird, a fourth, and a fifth embodiment of the invention;

FIG. 18 illustrates the internal structure of the horizontal variationpattern detector in FIG. 17 in the third and fourth embodiments;

FIG. 19 is a graph illustrating the operation of the comparators in FIG.18 in the third embodiment;

FIGS. 20A, 20B, and 20C are spatial waveform diagrams illustrating theoperation of the third embodiment;

FIG. 21 illustrates interpolation in the third embodiment at athree-pixel edge;

FIG. 22 illustrates the internal structure of the zoom processor in avariation of the third embodiment;

FIG. 23 is a graph illustrating the operation of the comparators in FIG.18 in the fourth embodiment;

FIG. 24 illustrates the internal structure of the horizontal variationpattern detector in FIG. 17 in the fifth embodiment;

FIG. 25A illustrates pixel data values in part of an image, with edgesdivided into leading, interior, and trailing parts;

FIG. 25B illustrates corresponding pixel data values after imageexpansion according to a sixth and a seventh embodiment of theinvention;

FIG. 26A illustrates pixel data values in part of an image, with edgesdivided into leading, trailing, and interior parts;

FIG. 26B illustrates corresponding pixel data values after imagereduction according to the sixth and seventh embodiments of theinvention;

FIG. 27 illustrates the internal structure of the zoom processor in thesixth embodiment;

FIGS. 28A, 28B, 28C, 28D, 28E, and 28F are spatial waveform diagramsillustrating the operation of the sixth embodiment;

FIG. 29 illustrates interpolation in the sixth embodiment at afour-pixel edge;

FIG. 30 illustrates the internal structure of the zoom processor in avariation of the sixth embodiment;

FIG. 31 illustrates the internal structure of the zoom processor in theseventh embodiment;

FIG. 32 illustrates the internal structure of the horizontal zoom ratiocontrol unit in FIG. 31;

FIGS. 33A, 33B, 33C, 33D, 33E, 33F, and 33G are spatial waveformdiagrams illustrating the operation of the seventh embodiment;

FIG. 34 illustrates the internal structure of the zoom processor in avariation of the seventh embodiment;

FIG. 35 illustrates the internal structure of the zoom processor in aneighth embodiment of the invention;

FIGS. 36A, 36B, 36C, 36D, and 36E are spatial waveform diagramsillustrating the operation of the eighth embodiment;

FIG. 37 illustrates interpolation in the eighth embodiment at athree-pixel edge;

FIG. 38 illustrates the internal structure of the zoom processor in avariation of the eighth embodiment;

FIG. 39 illustrates the internal structure of the zoom processor in aninth embodiment of the invention;

FIGS. 40A, 40B, 40C, 40D, 40E, 40F, and 40G are spatial waveformdiagrams illustrating the operation of the ninth embodiment;

FIG. 41 illustrates the internal structure of the zoom processor in atenth embodiment of the invention;

FIG. 42 illustrates the location of an interpolation point in the tenthembodiment;

FIG. 43 illustrates interpolation at the interpolation point in FIG. 42;

FIG. 44 illustrates the internal structure of the zoom processor in avariation of the tenth embodiment;

FIG. 45 illustrates the internal structure of the zoom processor inanother variation of the tenth embodiment;

FIG. 46 is a block diagram illustrating an image display apparatusaccording to an eleventh embodiment of the invention; and

FIG. 47 is a flowchart illustrating the operation of a twelfthembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention will be described with reference to theattached drawings, in which like parts are indicated by like referencecharacters.

The word ‘zoom’ will be used below as a generic term denoting any changein the number of pixels representing an image, or part of an image.Zooming may either increase the number of pixels, thereby eitherexpanding the image, or decrease the number of pixels, thereby reducingthe image. The ratio of the number of pixels after zooming to the numberbefore zooming will be referred to as the zoom ratio. An image may bezoomed horizontally, vertically, or in both directions. Horizontalzooming is performed by interpolation as illustrated in FIG. 2; verticalzooming is performed by similar interpolation in the vertical direction.

The first ten embodiments relate to the image display apparatus shown inFIG. 6. The apparatus comprises input terminals 1, 2 for an image signaland a synchronization (sync) signal. The image signal passes through, insequence, an analog-to-digital converter (ADC) 3, an image pre-processor4, a memory 5, a zoom processor 6, an image post-processor 7, and adigital-to-analog converter (DAC) 8; the processed image is thendisplayed by a display device 9. The operation of these componentelements is controlled by a control unit 10, which receives thesynchronization signal.

The input image signal is, for example, a composite video signal withluminance and chrominance components. Alternatively, the luminance andchrominance signals may be input separately, so that the input imagesignal comprises a pair of separate video input signals. The input imagesignal may also be a trio of signals for three primary colors (normallyred, blue, and green).

The control unit 10 uses the synchronization signal to generate asampling clock signal for the analog-to-digital converter 3, and togenerate clock signals, timing signals, and other control signals forthe other component elements.

The analog-to-digital converter 3 samples the input image signal insynchronization with the sampling clock, thereby converting the inputimage signal to digital data.

The image pre-processor 4 converts the digital data from the inputformat to the format used by the zoom processor 6, if these formatsdiffer. For example, the image pre-processor 4 may convert a compositevideo signal to separate luminance and chrominance data, or to separatedata for the three primary colors. The image pre-processor 4 may alsoprocess the data to adjust the brightness or contrast of the image. Theprocessing performed by the image pre-processor 4 is independent of thezooming process that will be performed by the zoom processor 6.

The memory 5 stores the pre-processed image data temporarily. The dataare written into the memory 5 in the usual image scanning sequence, eachhorizontal line being scanned from left to right, and the scanning linesbeing taken in order from top to bottom. The memory 5 should have enoughcapacity to store data for several scanning lines (at least two). Dataare read out from the memory 5 at timings determined according to therequirements of the display device 9, not necessarily in synchronizationwith the sampling clock, and not necessarily in the same order aswritten. The image data read from the memory 5 become the input data(Pi) of the zoom processor 6.

The zoom processor 6, which constitutes the invented image-processingapparatus, detects brightness-level changes in the input image data Pi,sets interpolation points by calculating zoom ratios, and generatesoutput image data Po by interpolation at the interpolation points,thereby zooming the image. Details will be given in the descriptions ofthe embodiments below.

The image post-processor 7 further processes the output image data Poby, for example, adjusting the brightness, contrast, or color saturationof the output image, or adjusting the luminance scale. These adjustmentsare independent of the zooming process.

The digital-to-analog converter 8 converts the post-processed image datato an analog signal or signals. The display device 9 displays the analogsignal(s) as an image on a screen, scanning the screen at a ratedetermined by the control unit 10.

FIGS. 7A and 7B illustrate image expansion and FIGS. 8A and 8Billustrate image reduction in the first five embodiments of theinvention. The horizontal axis in these drawings could correspond toeither the horizontal or the vertical direction in the image. Thehorizontal direction will be assumed in the following discussion, so thehorizontal axis represents a horizontal row of pixels. The vertical axisrepresents the pixel values, i.e., the pixel brightness levels.

The expansion method used in the first five embodiments can be describedgenerally as follows. The zoom processor 6 detects brightness-levelvariations in the input row of pixels (FIG. 7A). The information thusobtained divides the row into uniform segments (a), in each of which thebrightness level remains uniform, and edges (b, c), at which thebrightness level changes. The edges are further divided into first andsecond parts, more specifically, into leading edge segments (b) andtrailing edge segments (c). A basic zoom ratio (n) greater than unity isapplied to the uniform segments (a). A variable zoom ratio greater thann is applied in the leading edge segments (b). A variable zoom ratioless than n is applied in the trailing edge segments (c). The image isthen expanded by generating new image data (FIG. 7B) according to thesezoom ratios.

The basic zoom ratio (n) may have an arbitrary value. A typical value is1.6, used for converting a 640-by-480-pixel image to a 1024-by-768-pixelimage.

The reduction method used in the first five embodiments can be describedas follows. The zoom processor 6 again detects brightness-levelvariations in the input row of pixels (FIG. 8A), dividing the row ofpixels into uniform segments (a), leading edge segments (b), andtrailing edge segments (c). A basic zoom ratio (n) less than unity isapplied in the uniform segments (a). A variable zoom ratio higher than nis applied in the leading edge segments (b). A variable zoom ratio lowerthan n is applied in the trailing edge segments (c). The image is thenreduced by generating new image data (FIG. 8B) according to these zoomratios.

The internal structure and operation of the zoom processor 6 will now bedescribed. In the descriptions that follow, the zoom processor 6performs both horizontal and vertical zooming.

Referring to FIG. 9, the zoom processor 6 in the first embodimentincludes a vertical interpolator 11, a vertical high-frequencyinformation detector 12, a vertical zoom ratio control unit 13, ahorizontal interpolator 14, a horizontal high-frequency informationdetector 15, and a horizontal zoom ratio control unit 16. These elementscomprise, for example, specialized arithmetic and logic circuits,detailed descriptions of which will be omitted to avoid obscuring theinvention with unnecessary detail.

The vertical high-frequency information detector 12 receives input imagedata Pi from the memory 5 and performs operations equivalent tomathematical differentiation, obtaining the first derivative vd1 andsecond derivative vd2 of the image data in the vertical direction. Thesederivatives furnish information about comparatively high spatialfrequencies in the vertical direction.

The vertical zoom ratio control unit 13 uses the vertical derivativesvd1, vd2 output by the vertical high-frequency information detector 12to determine a vertical zoom ratio vc1, and supplies this zoom ratio vc1to the vertical interpolator 11.

The vertical interpolator 11 uses the supplied vertical zoom ratio vc1to generate vertically zoomed image data Pv from the input data Pi.

The horizontal high-frequency information detector 15 performshorizontal differentiation on the vertically zoomed image data Pv toobtain a horizontal first derivative hd1 and second derivative hd2.These derivatives furnish information about comparatively high spatialfrequencies in the horizontal direction.

The horizontal zoom ratio control unit 16 uses the horizontalderivatives hd1, hd2 output by the horizontal high-frequency informationdetector 15 to determine a horizontal zoom ratio hc1, and supplies thishorizontal zoom ratio hc1 to the horizontal interpolator 14.

The horizontal interpolator 14 uses the supplied horizontal zoom ratiohc1 to generate output image data Po from the vertically zoomed imagedata Pv.

In the zoom processor 6, the vertical high-frequency informationdetector 12 and horizontal high-frequency information detector 15function as first processing units, the vertical zoom ratio control unit13 and horizontal zoom ratio control unit 16 function as secondprocessing units, and the vertical interpolator 11 and horizontalinterpolator 14 function as third processing units.

Though not shown in the drawings, it is possible to insert a verticallow-pass spatial filter between the memory 5 and the verticalhigh-frequency information detector 12, to remove noise from the inputimage data Pi, so that the noise will not disturb the zoom ratio. Ifthis is done, the vertical low-pass spatial filter and verticalhigh-frequency information detector 12 together constitute a firstprocessing unit. The vertical interpolator 11 continues to receive theunaltered image data Pi read from the memory 5.

Similarly, a horizontal low-pass spatial filter may be inserted betweenthe vertical interpolator 11 and the horizontal high-frequencyinformation detector 15. In this case the horizontal low-pass spatialfilter and the horizontal high-frequency information detector 15together constitute a first processing unit. The horizontal low-passspatial filter may be provided in addition to or instead of the verticallow-pass spatial filter.

In place of a low-pass spatial filter, a bandpass spatial filter may beinserted between the memory 5 and vertical high-frequency informationdetector 12, and/or between the vertical interpolator 11 and horizontalhigh-frequency information detector 15. Similar noise-reduction effectsare obtained.

Alternatively, a coring unit with a dead band may be inserted betweenthe vertical high-frequency information detector 12 and vertical zoomratio control unit 13, or between the horizontal high-frequencyinformation detector 15 and horizontal zoom ratio control unit 16, toremove information about minor variations that are likely to be due tonoise.

The horizontal zooming operation will now be described in more detail.

FIG. 10A shows an example of brightness levels in a horizontal line inthe image data Pv output by the vertical interpolator 11. FIG. 10B showsthe corresponding horizontal first derivative hd1 output by thehorizontal high-frequency information detector 15, calculated as, forexample, differences in brightness level between mutually adjacentpixels. FIG. 10C shows the corresponding horizontal second derivativehd2 output by the horizontal high-frequency information detector 15. Thehorizontal zoom ratio control unit 16 calculates the horizontal zoomratio hc1 by the following equation, in which n is the basic zoom-ratioparameter mentioned earlier, and k is an arbitrary positive constantparameter.

hc1=n+(k×hd1×hd2)

The variable zoom ratio (hc1) obtained with this equation is shown inFIG. 10D. As already noted, the zoom ratio hc1 is equal to n in thesegments (a) of uniform brightness level, is greater than n in theleading edge segments (b), and is less than n in the trailing edgesegments (c). The average value of hc1 over the entire horizontal lineis equal to n. If AVE(x) indicates the average value of x over onehorizontal line, then:

AVE(hc1)=n

FIG. 10E depicts expanded image data Po obtained by interpolation withthis varying horizontal zoom ratio hc1.

Similar operations are performed in the vertical direction to generate avarying vertical zoom ratio vc1. The average of this varying verticalzoom ratio vc1 is again equal to n.

FIG. 11 illustrates the generation of output image data at an edgecomprising three pixels (p1, p2, p3) in the image data Pv, when thebasic zoom ratio (n) applied in uniform segments (a) is three. Outputpixels are generated by interpolation at seven points (q11 to q17) usingthe type of linear filter response characteristic illustrated in FIG. 2.If the distance between p1 and p2 is taken to be unity, then the zoomratio hc1 is the reciprocal of the spacing between adjacentinterpolation points. This spacing can be seen to vary according to thecurve in FIG. 10D, becoming shortest between q12 and q13, where the zoomratio is highest, and longest between q15 and q16, where the zoom ratiois lowest.

Approximately speaking, the distance between points q11 and q12 is thereciprocal of the zoom ratio at q12, the distance between q12 and q13 isthe reciprocal of the zoom ratio at q13, and so on. New pixels can thusbe generated at points given by the cumulative sum of the reciprocals ofthe zoom ratios. Specifically, the horizontal zoom ratio control unit 16calculates the zoom ratio hc1 at the conventional evenly-spacedinterpolation points (not visible), and the reciprocals (1/hc1) of thecalculated hc1 values are cumulatively summed to set the interpolationpoints (q11 to q17) at which new pixels are actually generated. Thisprocess will be illustrated in the eighth embodiment.

Thus by calculating a variable zoom ratio hc1, the horizontal zoom ratiocontrol unit 16 sets interpolation points (e.g., q11 to q17) withvariable spacing in each horizontal line. A basic spacing value (1/n) isused in uniform segments (a). The spacing is decreased in leading edgesegments (b), and increased in trailing edge segments (c).

The interpolation points given as cumulative sums of the 1/hc1 valuesdetermine the filter coefficients used to generate the output pixelvalues, and determine the vertically zoomed pixels (p1, p2, etc.) towhich the filter coefficients are applied. If the vertically zoomedpixels (p1, p2, etc.) are stored in the memory 5, the horizontalinterpolator 14 converts the cumulative sums of 1/hc1 to addresses inthe memory 5.

The filter coefficients may be pre-calculated and stored in a look-uptable in the memory 5 (or another memory unit, not shown in thedrawings). In that case, the horizontal interpolator 14 also convertsthe cumulative sums of the 1/hc1 to addresses in the look-up table.

When the output image is displayed, the interpolated pixels are evenlyspaced, as indicated by s11 to s17 at the bottom of FIG. 11. Thus theleading edge segment (b), which has a zoom ratio higher than n, isexpanded into a wider segment than the trailing edge segment (c), whichhas a zoom ratio lower than n. As a result, the variation in brightnesslevel is not distributed evenly from pixel s11 to s17 in the expandededge; most of the variation occurs in the trailing half of the expandededge, from pixel s14 to pixel s17. The expanded edge thus appearssharper than it would appear if it had been expanded by the conventionaluniform zoom ratio.

When an image is reduced, conversely, trailing edge segments are reducedby more than the basic zoom ratio, and leading edge segments are reducedby less than the basic zoom ratio. The conventional problem of pixeldropout is therefore mitigated in the leading edge segments, in whichthe number of pixels is not reduced so much.

Compared with conventional methods, the first embodiment is accordinglyable to expand and reduce images by arbitrary zoom ratios with lessnoticeable loss of edge sharpness when an image is expanded, and lessnoticeable pixel drop-out when an image is reduced, and withoutproducing noticeable discontinuities, since the interpolation spacingvaries in a continuous manner.

The parameter k employed in calculating the variable zoom ratio hc1 canbe used to control edge sharpness in the output image, larger values ofk leading to sharper edges. This technique can be used to control edgesharpness even when the basic zoom ratio (n) is unity, so that the imageas a whole is not expanded or reduced.

The parameters (n, k) used for horizontal zooming need not be the sameas the parameters (n, k) used for vertical zooming. For example, thebasic zoom-ratio parameter can be set to two (n=2) vertically and one(n=1) horizontally to convert from interlaced to progressive scanning byinterpolating new scanning lines, with independent control of edgesharpness in the vertical and horizontal directions.

The variable zoom ratio can be calculated by more complex schemes thansimply adding the product of the first derivative, the secondderivative, and the parameter k to the basic zoom ratio n. For example,a positive upper limit and a negative lower limit may be placed on theproduct, or a non-linear function may be applied as a substitute formultiplication by k.

Similarly, the filter response characteristic may be non-linear, insteadof linear as shown in FIG. 11.

FIG. 9 showed vertical zooming being performed before horizontalzooming, but horizontal zooming may be performed before verticalzooming.

Referring to FIG. 12, horizontal zooming and vertical zooming may becarried out simultaneously. The zoom processor 6 then comprises atwo-dimensional interpolator 17, a high-frequency information detector18, and a two-dimensional zoom ratio control unit 19. The high-frequencyinformation detector 18 detects both horizontal and vertical variationsin the input image data and calculates a two-dimensional firstderivative d1 and second derivative d2. The two-dimensional zoom ratiocontrol unit 19 calculates a two-dimensional zoom ratio c1 from thesederivatives d1 and d2, by a two-dimensional version of the calculationdescribed above. The two-dimensional interpolator 17 uses thetwo-dimensional zoom ratio c1 and the input image data Pi fortwo-dimensional interpolation, thereby performing both horizontal andvertical zooming. Further details will be given in the description ofthe tenth embodiment.

FIG. 13 illustrates the zoom processor 6 in a second embodiment of theinvention. The vertical interpolator 11 and horizontal interpolator 14are similar to the corresponding elements in the first embodiment.

The vertical high-frequency information detector 20 receives the inputimage data Pi and detects high spatial frequency information by takingdifferences between vertically adjacent pixels, performing an operationthat generates the first derivative vd1 of the image data in thevertical direction. The vertical zoom ratio control unit 21 uses thisfirst derivative vd1 to determine a vertical zoom ratio vc2. Thevertical interpolator 11 uses the vertical zoom ratio vc2 to generatevertically zoomed image data Pv from the input image data Pi.

The horizontal high-frequency information detector 22 receives thevertically zoomed image data Pv and detects high spatial frequencyinformation by taking differences between horizontally adjacent pixels,generating the first derivative hd1 of the image data Pv in thehorizontal direction. The horizontal zoom ratio control unit 23 usesthis first derivative hd1 to determine a horizontal zoom ratio hc2. Thehorizontal interpolator 14 uses the horizontal zoom ratio hc2 togenerate output image data Po from the vertically zoomed image data Pv.

Horizontal zooming in the second embodiment is illustrated in FIGS. 14Ato 14D. FIG. 14A shows an example of brightness levels in a horizontalline in the image data Pv output by the vertical interpolator 11. FIG.14B shows the corresponding horizontal first derivative hd1 output bythe horizontal high-frequency information detector 22. FIG. 14C showsthe horizontal zoom ratio hc2 calculated by the horizontal zoom ratiocontrol unit 23. At an edge (b, c), the horizontal zoom ratio controlunit 23 calculates hc2 by the following equations, in which n is thebasic zoom ratio, k is an arbitrary positive constant parameter, absdenotes absolute value, and r is the distance across the edge,normalized so that the width of the edge (b+c) is unity.

hc2=n+(k×abs(hd1)) if 0.0≦r<0.5

hc2=n−(k×abs(hd1)) if 0.5≦r<1.0

As illustrated in FIG. 14C, the average value of the zoom ratio over theentire edge is the basic zoom ratio (n).

AVE(hc2)=n

FIG. 14D depicts expanded output image data Po obtained by interpolationwith the variable zoom ratio hc2.

FIG. 15 shows another example, in which the edge comprises only twopixels (p4, p5) in the vertically zoomed image data Pv. The edge isstill divided into a leading segment (b) and a trailing segment (b). Inthis example, the first derivative (hd1) is constant over both segments.The zoom ratio (hc2) is greater than n in the leading segment (b), andless than n in the trailing segment (c). The interpolation points of theoutput pixels (q20 to q25) are spaced more closely in the leadingsegment (b) than in the trailing segment (c). As in the firstembodiment, if the distance between p4 and p5 is taken to be unity, thenthe spacing between the interpolation points is equal to the reciprocalof the zoom ratio. When displayed, the interpolated pixels (s20 to s25)are evenly spaced.

Vertical zooming is performed in the same way as horizontal zooming. Thehorizontal zoom ratio and vertical zoom ratio are mutually independent,and different values of the parameter k may be used in the horizontaland vertical directions.

The second embodiment has the same general effect as the firstembodiment. Loss of edge sharpness is mitigated by concentrating more ofthe edge variation into trailing edge segments when an image isexpanded. Pixel drop-out is mitigated by keeping more pixels in leadingedge segments when an image is reduced.

In a variation of the second embodiment, the sign of the firstderivative is switched from positive to negative at an arbitrary point i(0<i<1) instead of at the midpoint (r=0.5). In this variation, the zoomratio is calculated as follows.

hc2=n+((1−i)×k×abs(hd1)) if 0≦r<i

hc2=n−(i×k×abs(hd1)) if i≦r<1.0

In another variation of the second embodiment, the first derivative iscalculated on the basis of three or more consecutive pixels, instead ofby simply taking the difference between two adjacent pixels. Theparameters r and i shown above may be normalized so that the distancebetween the outermost pixels referenced in calculating the firstderivative is equal to unity.

Horizontal zooming and vertical zooming may also be carried outsimultaneously in the second embodiment. The zoom processor 6 then hasthe structure shown in FIG. 16, comprising a two-dimensionalinterpolator 17, a high-frequency information detector 24, and atwo-dimensional zoom ratio control unit 25. The high-frequencyinformation detector 24 detects both horizontal and vertical variationsin the input image data and calculates a two-dimensional firstderivative d1. The two-dimensional zoom ratio control unit 25 calculatesa two-dimensional zoom ratio c2 from the two-dimensional firstderivative d1, using the equations given above in each dimension. Thetwo-dimensional interpolator 17 uses the two-dimensional zoom ratio c2for two-dimensional (horizontal and vertical) interpolation. Furtherdetails will be given in the tenth embodiment.

FIG. 17 illustrates the zoom processor 6 in a third embodiment of theinvention. The vertical interpolator 11 and horizontal interpolator 14are similar to the corresponding elements in the first embodiment.

The vertical variation pattern detector 26, operating as a firstprocessor, receives the input image data Pi from the memory 5, detectshigh spatial frequency information by detecting patterns of differencesbetween vertically adjacent pixels, and generates vertical variationpattern information vp1. The vertical zoom ratio control unit 27 usesthe vertical variation pattern information vp1 to determine a verticalzoom ratio vc3. The vertical interpolator 11 uses the vertical zoomratio vc3 to generate vertically zoomed image data Pv from the inputimage data Pi.

The horizontal variation pattern detector 28 receives the verticallyzoomed image data Pv, detects high spatial frequency information bydetecting patterns of differences between horizontally adjacent pixels,and generates horizontal variation pattern information hp1. Thehorizontal zoom ratio control unit 29 uses the horizontal variationpattern information hp1 to determine a horizontal zoom ratio hc3. Thehorizontal interpolator 14 uses the horizontal zoom ratio hc3 togenerate output image data Po from the vertically zoomed image data Pv.

As explained in the first embodiment, a vertical low-pass filter orbandpass filter may be inserted between the memory 5 and the verticalvariation pattern detector 26, to remove noise from the image datasupplied to the vertical variation pattern detector 26, and a horizontallow-pass filter or bandpass filter may be inserted between the verticalinterpolator 11 and the horizontal variation pattern detector 28, toremove noise from the vertically zoomed image data Pv.

FIG. 18 shows the internal structure of the horizontal variation patterndetector 28, which comprises a pair of delay units 30, 31, a pair ofcomparators 32, 33, and an image data comparison result processor 34.

The image data Pv output from the vertical interpolator 11 are suppliedas pixel data Pd0 to the comparator 32, and are also supplied to delayunit 30. Delay unit 30 stores the received data Pd0 for one pixelinterval, and supplies the stored data as pixel data Pd1 to thecomparators 32, 33 and delay unit 31. Delay unit 31 stores the receiveddata Pd1 for one pixel interval, and supplies the stored data as pixeldata Pd2 to comparator 33.

Comparator 32 thus receives pixel data delayed by zero pixels (Pd0) andone pixel (Pd1) with respect to the image data Pv, while comparator 33receives pixel data delayed by one pixel (Pd1) and two pixels (Pd2) withrespect to the image data Pv. The three pixel values supplied to the twocomparators 32, 33 correspond to three consecutive pixels (Pd2, Pd1,Pd0) in one horizontal scanning line of the image. Comparator 32receives the third pixel (Pd0) as its first input (a) and the secondpixel (Pd1) as its second input (b), compares the two input values, andgenerates a comparison result hcomp2. Comparator 33 receives the secondpixel (Pd1) as its first input (a) and the first pixel (Pd2) as itssecond input (b), compares the two input values, and generates acomparison result hcomp1.

The comparators 32, 33 operate as indicated in FIG. 19. The horizontalaxis represents the first input value (a) on a scale from zero to twohundred fifty-five, which is suitable if the input values are eight-bitvalues. The vertical axis represents the second input value (b) on asimilar scale from zero to two hundred fifty-five. If the second input(b) exceeds the first input (a) by more than a predetermined quantity(d), the comparison result has a first value, indicated by a plus sign(+) in the drawing, meaning that the second input is larger. If thefirst input (a) exceeds the second input (b) by more than thepredetermined quantity (d), the comparison result has a second value,indicated by a minus sign (−) in the drawing, meaning that the firstinput is larger. If the difference between the two input values (a andb) is equal to or less than the predetermined quantity (d), thecomparison result has a third value (0), meaning that the two inputs areapproximately equal. The comparators 32, 33 thus operate with a deadband from −d to +d, described below as a dead band of ±d, generatinginformation indicating the polarity (positive, negative, or zero) of thedifference between the values of adjacent pixels (zero indicating thedead band).

The purpose of the ±d dead band is to reduce the effect of noise in theimage data.

The image data comparison result processor 34 generates the horizontalvariation pattern information hp1 by combining the comparison resultshcomp1 and hcomp2. The horizontal variation pattern information hp1 isaccordingly a digital signal with nine values, identified as A to J inTable 1. Table 1 also lists the zoom ratios hc3 output by the horizontalzoom ratio control unit 29 for each pattern, n being the basic zoomratio and α being an arbitrary positive quantity less than n (0<α<n).

Each pattern represents a pattern of variation of three horizontallyconsecutive pixels. The listed zoom ratio hc3 is applied at the positionof the second of the three pixels.

TABLE 1 hcomp1 hcomp2 hp1 hc3 (0) (0) A n (0) (+) B n + α (0) (−) C n +α (+) (0) D n − α (−) (0) E n − α (+) (+) F n (−) (−) G n (+) (−) H n(−) (+) J n

In pattern A, there are substantially no pixel-to-pixel variations (nonegreater than ±d), so the basic zoom ratio (n) is applied.

In patterns B and C, there is substantially no variation between thefirst two pixel values, but the third pixel value is significantlyhigher or lower than the second pixel value. These patterns correspondto leading edge segments. An increased zoom ratio (n+α) is applied.

In patterns D and E, there is substantially no variation between thelast two pixel values, but the first pixel value is significantly higheror lower than the second pixel value. These patterns correspond totrailing edge segments. A decreased zoom ratio (n−α) is applied.

In patterns F and G, the same type of variation appears between thefirst two pixels as between the last two pixels, either an increase inboth cases, or a decrease in both cases. The basic zoom ratio (n) isapplied.

In patterns H and J, opposite variations appear between the first twopixels and the last two pixels, the variation being an increase in onecase and a decrease in the other case. The basic zoom ratio (n) isapplied.

FIG. 20A shows an example of pixel data values in a horizontal line inthe vertically zoomed image data Pv, and the corresponding patterninformation hp1 output by the horizontal variation pattern detector 28.FIG. 20B shows the zoom ratio hc3 output by the horizontal zoom ratiocontrol unit 29. The zoom ratios at positions between the pixels in theimage data Pv are calculated by linear interpolation. As in the firstembodiment, the horizontal scanning line can be divided into three typesof segments: uniform segments (a), in which the basic zoom ratio (n) isapplied; leading edge segments (b), in which the zoom ratio variesbetween n and n+α; and trailing edge segments (c), in which the zoomratio varies between n and n−α. FIG. 20C shows the appearance of thesethree types of segments (a, b, c) in expanded data output by thehorizontal interpolator 14.

FIG. 21 illustrates horizontal zooming in the third embodiment at anedge comprising three horizontally consecutive pixels (p1, p2, p3) inthe vertically zoomed image data Pv, when the basic zoom ratio (n)applied in uniform segments (a) is three. Output image data aregenerated at seven interpolation points (q11 to q17), using the type oflinear filter response characteristic illustrated in FIG. 2. Asexplained in the first embodiment, if the distance between p1 and p2 istaken to be unity, then the zoom ratio is the reciprocal of the distancebetween the interpolation points, so the positions of the interpolationpoints can be obtained by cumulative summation of the reciprocals of thezoom ratios output by the horizontal zoom ratio control unit 29. Whendisplayed, the output pixels (s11 to s17) are equally spaced.

A comparison of FIG. 21 with FIG. 11 shows that the third embodimentprovides much the same effect as the first embodiment. Leading edgesegments (b) are zoomed with higher zoom ratios than trailing edgesegments (c), and the average value of the zoom ratio over an entirehorizontal scanning line is equal to the basic zoom ratio (n).

AVE(hc3)=n

In the third embodiment, the parameter α can be adjusted to control edgesharpness in the output image. Larger values of α lead to sharper edges.Edge sharpness can be controlled in this way even when the basic zoomratio n is equal to unity.

The basic zoom ratio n and parameter α used in the horizontal directioncan be set independently of the basic zoom ratio n and parameter α usedin the vertical direction.

Referring to FIG. 22, horizontal zooming and vertical zooming may alsobe carried out simultaneously in the third embodiment. The zoomprocessor 6 then comprises a two-dimensional interpolator 17, avariation pattern detector 35, and a two-dimensional zoom ratio controlunit 36. The variation pattern detector 35 detects both horizontal andvertical variations in the input image data and generates both verticalvariation pattern information vp1 and horizontal variation patterninformation hp1. The two-dimensional zoom ratio control unit 36calculates a two-dimensional zoom ratio c3 from the supplied patterninformation vp1 and hp1. The two-dimensional interpolator 17 uses thetwo-dimensional zoom ratio c3 for two-dimensional (horizontal andvertical) interpolation. A detailed description of the calculation ofvp1, hp1, and c3 will be omitted, since the process is analogous to thecalculation of hp1 and hc3 summarized in Table 1, but thetwo-dimensional interpolation process will be illustrated in the tenthembodiment.

In a fourth embodiment of the invention, the zoom ratio is adjustedaccording to both the polarity and magnitude of the pixel-to-pixelvariations. The fourth embodiment has the same hardware configuration asthe third embodiment, illustrated in FIGS. 17 and 18, but in relation tohorizontal zooming, for example, the comparators 32, 33 operate asillustrated in FIG. 23, generating comparison results (hcomp1, hcomp2)with five possible values.

As in the third embodiment, one comparison result (0) indicates that thefirst and second inputs (a and b) do not differ by more than apredetermined positive quantity d.

Another comparison result (+) indicates that the second input (b)exceeds the first input (a) by more than d, but does not exceed twicethe first input (a) by more than d.

Another comparison result (2+) indicates that the second input (b)exceeds twice the first input (a) by more than d.

Another comparison result (−) indicates that the first input (a) exceedsthe second input (b) by more than d, but that half the first input doesnot exceed the second input (b) by more than d.

Another comparison result (2−) indicates that half the first input (a)exceeds the second input (b) by more than d.

These results are summarized by the inequalities at the bottom of FIG.23.

From these comparison results, the image data comparison resultprocessor 34 generates the pattern information (A to J) described in thethird embodiment, and four additional patterns (B+, C−, D+, E−). Whenthe image data comparison result processor 34 outputs one of theseadditional patterns, the horizontal zoom ratio control unit 29 adjuststhe zoom ratio (n±α) used in the third embodiment by a further quantityβ. This further quantity β may be positive or negative, but its absolutevalue is less than the difference between n and α. Accordingly, thefollowing condition is satisfied.

n−α−ABS(β)>0

Table 2 summarizes the comparator outputs (hcomp1, hcomp2), thehorizontal variation pattern information (hp1), and the zoom ratio (hc3)generated by the horizontal variation pattern detector 28 and horizontalzoom ratio control unit 29 in the fourth embodiment. As in the thirdembodiment, the horizontal variation pattern information (hp1) describesthe pattern of variation of the levels of three horizontally consecutivepixels, and the zoom ratio (hc3) is applied at the position of thesecond of these three pixels.

In pattern A, there are substantially no pixel-to-pixel variations (nonegreater than ±d), and the basic zoom ratio (n) is applied.

In patterns B and C, there is substantially no variation between thefirst two pixel values, but the third pixel value is moderately higheror lower than the second pixel value. These patterns correspond tocomparatively gentle leading edge segments. An increased zoom ratio(n+α) is applied.

In patterns B+ and C−, there is substantially no variation between thefirst two pixel values, but the third pixel value is markedly higher orlower than the second pixel value. These patterns correspond tocomparatively sharp leading edge segments. The increased zoom ratio(n+α) is further adjusted (to n+α+β).

TABLE 2 hcomp1 hcomp2 hp1 hc3 (0) (0) A n (0) (+) B n + α (0) (2+) B+n + α + β (0) (−) C n + α (0) (2−) C− n + α + β (+) (0) D n − α (2+) (0)D+ n − α − β (−) (0) E n − α (2−) (0) E− n − α − β (+) (+) F n (2+) (+)(+) (2+) (2+) (2+) (−) (−) G n (2−) (−) (−) (2−) (2−) (2−) (+) (−) H n(2+) (−) (+) (2−) (2+) (2−) (−) (+) J n (2−) (+) (−) (2+) (2−) (2+)

In patterns D and E, there is substantially no variation between thelast two pixel values, but the first pixel value is moderately higher orlower than the second pixel value. These patterns correspond tocomparatively gentle trailing edge segments. A decreased zoom ratio(n−α) is applied.

In patterns D+ and E−, there is substantially no variation between thelast two pixel values, but the first pixel value is markedly higher orlower than the second pixel value. These patterns correspond tocomparatively sharp trailing edge segments. The decreased zoom ratio(n−α) is further adjusted (to n−α−β).

In patterns F and G, the same type of variation appears between thefirst two pixels as between the last two pixels, either an increase inboth cases, or a decrease in both cases. The basic zoom ratio (n) isapplied, regardless of the magnitude of the variations.

In patterns H and J, opposite variations appear between the first twopixels and the last two pixels, an increase in one case and a decreasein the other case. The basic zoom ratio (n) is applied, regardless ofthe magnitude of the variations.

If the parameter β is positive, then the zoom ratio is further increasedat comparatively sharp leading edge segments, and further reduced atcomparatively sharp trailing edge segments. As a result, thesecomparatively sharp trailing edge segments appear sharper in the imagedisplayed by the fourth embodiment than in the image displayed by thethird embodiment.

If the parameter β is negative, then the zoom ratio is made lower atcomparatively sharp leading edge segments than at comparatively gentleleading edge segments, and is made higher at comparatively sharptrailing edge segments than at comparatively gentle trailing edgesegments. As a result, comparatively sharp leading edge segments appearsharper in the image displayed by the fourth embodiment than in theimage displayed by the third embodiment.

Vertical zooming in the fourth embodiment is controlled in the same wayas horizontal zooming, using parameters α and β and a basic zoom-ratioparameter n. The parameters (n, α, β) used in vertical zooming need notbe identical to the parameters (n, α, β) used in horizontal zooming.

The two parameters α and β employed in the fourth embodiment permit moreflexible control over edge sharpness than the single parameter α used inthe third embodiment. In particular, the value of β can be chosen toavoid over-enhancement or under-enhancement of edges. The fourthembodiment is particularly effective when a single pair of parametervalues (α and β) is used for all images. However, it is also possible toprovide a selection of parameter values (α and β), to fine-tune thezooming process.

In a variation of the fourth embodiment, interpolation is carried outtwo-dimensionally, as in FIG. 22.

Whereas the pattern information in the third and fourth embodimentsrepresented patterns of variation in the brightness levels of threeconsecutive pixels, in a fifth embodiment, the pattern informationrepresents patterns of variation in the brightness levels of fiveconsecutive pixels. Accordingly, the fifth embodiment has the generalconfiguration shown in FIGS. 6 and 17, but the horizontal variationpattern detector 28, for example, has the internal structure shown inFIG. 24, comprising four delay units 37, 38, 39, 40, four comparators41, 42, 43, 44, and an image data comparison result processor 45.

The delay units 37, 38, 39, 40 are coupled in series and generate imagedata Pd1, Pd2, Pd3, Pd4 delayed by one, two, three, and four pixelintervals, respectively, with respect to the vertically zoomed imagedata Pv (Pd0).

Each of the comparators 41, 42, 43, 44 is similar to the comparators inthe third embodiment, having two input terminals (a and b) andgenerating a comparison result with three possible values (+, 0,−).These three values indicate whether the second input is greater than,approximately equal to, or less than the first input, the approximatelyequal result (0) corresponding to a dead band of ±d. Comparator 44receives Pd3 and Pd4, and generates a comparison result hcomp0.Comparator 43 receives Pd2 and Pd3, and generates a comparison resulthcomp1. Comparator 42 receives Pd2 and Pd3, and generates a comparisonresult hcomp2. Comparator 41 receives Pd0 and Pd1, and generates acomparison result hcomp3.

The image data comparison result processor 45 receives the comparisonresults from the comparators 41, 42, 43, 44, and generates horizontalvariation pattern information hpl as indicated in Table 3. An X in Table3 indicates a don't-care value. For example, pattern information A isoutput whenever hcomp1 and hcomp2 both indicate approximate equality(0), regardless of the values of hcomp0 and hcomp3. The patterninformation includes patterns A to J as described in the thirdembodiment, and four new patterns (BF, CG, DF, EG).

The horizontal zoom ratio control unit 29 generates a corresponding zoomratio hc3, also indicated in Table 3. This zoom ratio is applied at theposition of the third pixel (Pd2) in the series of five pixels (Pd0 toPd4). For patterns A to J, the horizontal zoom ratio control unit 29generates the zoom ratios described in the third embodiment (n, n+α,n−α). For the four new patterns, the horizontal zoom ratio control unit29 adjusts the zoom ratio (n±α) used in the third embodiment by afurther quantity γ. This further quantity γ may be positive or negative,but its absolute value must satisfy the same condition as satisfied bythe parameter β in the fourth embodiment, that is:

n−α−ABS(γ)>0

TABLE 3 hcomp0 hcomp1 hcomp2 hcomp3 hp1 hc3 X (0) (0) X A n X (0) (+)(−) B n + α (0) X (0) (+) (+) BF n + α + γ X (0) (−) (−) CG n + α + γ X(0) (−) (0) C n + α (+) (−) (+) (0) X D n − α (0) (+) (+) (0) X DF n − α− γ (−) (−) (0) X EG n − α − γ (0) (−) (0) X E n − α (+) X (+) (+) X F nX (−) (−) X G n X (+) (−) X H n X (−) (+) X J n

In pattern A, there are substantially no pixel-to-pixel variations (nonegreater than ±d) among the second, third, and fourth pixels, and thebasic zoom ratio (n) is applied.

In patterns B and C, there is substantially no variation between thesecond and third pixels. The fourth pixel value is significantly higheror lower than the third pixel value, but this same variation is not seenbetween the fourth and fifth pixels. An increased zoom ratio (n+α) isapplied.

In patterns BF and CG, the variation seen in the third and fourth pixelsin pattern B or C is also seen in the fourth and fifth pixels. Theincreased zoom ratio (n+α) is further adjusted (to n+α+γ).

In patterns D and E, there is substantially no variation between thethird and fourth pixels. The second pixel value is significantly higheror lower than the third pixel value, but this same variation is not seenbetween the first and second pixels. A decreased zoom ratio (n−α) isapplied.

In patterns DF and EG, the variation seen in the second and third pixelsin pattern D or E is also seen in the first and second pixels. Thedecreased zoom ratio (n−α) is further adjusted (to n−α−γ).

In patterns F and G, the same type of variation appears between thesecond and third pixels and between the third and fourth pixels, eitheran increase in both cases, or a decrease in both cases. The basic zoomratio (n) is applied, regardless of the values of the first and fifthpixels.

In patterns H and J, opposite variations appear between the second andthird pixels and the third and fourth pixels, an increase in one caseand a decrease in the other case. The basic zoom ratio (n) is applied,regardless of the values of the first and fifth pixels.

The fifth embodiment distinguishes between brightness-level variationsthat continue over three or more consecutive pixels, and variations thatoccur at only one pair of pixels. If the parameter γ is positive, thenthe zoom ratio is altered more for the former variations than for thelatter variations. If the parameter γ is negative, then the zoom ratiois altered more for the latter variations than for the formervariations.

Vertical zooming in the fifth embodiment is controlled in the same wayas horizontal zooming, using parameters α and γ. The zoom ratio andother parameters (n, α, γ) used in vertical zooming need not beidentical to the zoom ratio and other parameters (n, α, γ) used inhorizontal zooming.

The two parameters α and γ employed in the fifth embodiment permit moreflexible control over edge sharpness than the single parameter α used inthe third embodiment, because the width of the edge is taken intoaccount. The fifth embodiment is particularly effective when a singlepair of parameter values (α and γ) is used for all images. However, itis also possible to provide a selection of parameter values (α and γ),to fine-tune the zooming process.

In a variation of the fifth embodiment, the comparators 41, 42, 43, 44generate comparison results with five possible values, as in the fourthembodiment, and the image data comparison result processor 45 determinesthe zoom ratio from the combination of these comparison results.

In another variation of the fifth embodiment, a two-dimensional zoomratio is calculated, and interpolation is carried out two-dimensionally.

Although the third, fourth, and fifth embodiments have been described asdividing an edge into leading and trailing segments, when a variationpersists over a sufficient number of consecutive pixels, the varyingportion is divided into three segments: a leading segment (e.g., patternB or C), an interior segment (e.g., pattern F or G), and a trailingsegment (e.g., pattern D or E). The zoom ratio is increased in theleading segment and reduced in the trailing segment. The basic zoomratio (n) is used in the interior segment.

A different way of processing interior segments will be shown in thesixth and seventh embodiments of the invention.

FIGS. 25A and 25B illustrate image expansion and FIGS. 26A and 26Billustrate image reduction in the sixth and seventh embodiments. Thehorizontal axis in these drawings corresponds to one direction, eitherthe horizontal or the vertical direction, in the image. The verticalaxis represents the data values (brightness levels) of a row or columnof pixels disposed in this direction.

In these embodiments, the input row or column of pixels (FIGS. 25A and26A) is divided into uniform segments (a) in which the brightness leveldoes not change, and edges (b, c). The edges are further divided intofirst, second, and third parts, more specifically into leading andtrailing segments (b) and interior segments (c). A leading or trailingsegment (b) is defined to be a segment in which the rate of change ofthe brightness level is itself changing. An interior segment (c) isdefined to be a segment in which the brightness level changes at asubstantially constant rate.

A basic zoom ratio (n) is applied in the uniform segments (a). Avariable zoom ratio higher than n is applied in the leading and trailingedge segments (b). A variable zoom ratio lower than n is applied in theinterior edge segments (c). The image is expanded or reduced bygenerating new image data (FIGS. 25B and 26B) at interpolation pointspositioned according to these zoom ratios. The average zoom ratio overthe entire row or column of pixels is equal to n.

The internal structure of the zoom processor 6 in the sixth embodimentis shown in FIG. 27. The vertical interpolator 11 and horizontalinterpolator 14 are similar to the corresponding elements in the firstembodiment.

The vertical high-frequency information detector 46 receives input imagedata Pi from the memory 5 and detects high spatial frequency informationby performing operations that generate the first derivative vd1 andthird derivative vd3 of the image data in the vertical direction. Thevertical zoom ratio control unit 47 uses these derivatives vd1, vd3 todetermine a vertical zoom ratio vc4. The vertical interpolator 11 usesthe vertical zoom ratio vc4 to generate vertically zoomed image data Pvfrom the received image data Pi.

The horizontal high-frequency information detector 48 receives thevertically zoomed image data Pv and detects high spatial frequencyinformation by performing operations that generate the first derivativehd1 and third derivative hd3 of the image data in the horizontaldirection. The horizontal zoom ratio control unit 49 uses thesederivatives hd1, hd3 to determine a horizontal zoom ratio hc4. Thehorizontal interpolator 14 uses the horizontal zoom ratio hc4 togenerate output image data Po from the vertically zoomed image data Pv.

As explained in the first embodiment, a vertical low-pass filter orbandpass filter may be inserted between the memory 5 and the verticalhigh-frequency information detector 46, to remove noise from the inputimage data Pi, and a horizontal low-pass filter or bandpass filter maybe inserted between the vertical interpolator 11 and the horizontalhigh-frequency information detector 48, to remove noise from thevertically zoomed image data Pv. Alternatively, a coring unit with adead band may be inserted between the vertical high-frequencyinformation detector 46 and vertical zoom ratio control unit 47, orbetween the horizontal high-frequency information detector 48 andhorizontal zoom ratio control unit 49.

FIG. 28A shows an example of the brightness levels in a horizontal linein the image data Pv output by the vertical interpolator 11. FIG. 28Bshows the corresponding horizontal first derivative hd1 output by thehorizontal high-frequency information detector 48. FIG. 28C shows thecorresponding horizontal second derivative, which is calculatedinternally by the horizontal high-frequency information detector 48.FIG. 28D shows the corresponding horizontal third derivative hd3 outputby the horizontal high-frequency information detector 48. The horizontalzoom ratio control unit 49 calculates the horizontal zoom ratio hc4 bythe following equation, in which n is the basic zoom ratio, and k is anarbitrary positive constant parameter.

hc4=n+(k×hd1×hd3)

The result of applying this equation is shown in FIG. 28E. As describedearlier, the zoom ratio hc4 is equal to n in the segments (a) of uniformbrightness level, is greater than n in the leading and trailing edgesegments (b), and is less than n in interior edge segments (c). Theaverage value of hc4 over the entire horizontal line is equal to n.

AVE(hc4)=n

FIG. 28F depicts expanded image data Po obtained by interpolation withthis varying zoom ratio hc4.

Similar operations are performed in the vertical direction to generate avarying vertical zoom ratio vc4. The average of this varying verticalzoom ratio vc4 is again equal to n.

FIG. 29 illustrates the generation of output image data Po at an edgecomprising four horizontally consecutive pixels (p1, p2, p3, p4) in thevertically zoomed image data Pv, when the basic zoom ratio (n) appliedin uniform segments (a) is three. New image data are generated at teninterpolation points (q11 to q20), using the type of linear filterresponse characteristic illustrated in FIG. 2. If the distance betweenp1 and p2 is taken to be unity, then the zoom ratio is the reciprocal ofthe distance between adjacent interpolation points. This distance can beseen to vary inversely to the height of the curve in FIG. 28E, beingrelatively short in the leading and trailing segments (b) and relativelylong in the interior segment (c).

As explained in the first embodiment, the interpolation points are setat positions equal to cumulative sums of the reciprocals of the zoomratios. That is, the filter coefficients used to generate the new pixelvalues, and the memory addresses of the pixels (p1, p2, etc.) to whichthe filter coefficients are applied, are obtained from these cumulativesums.

When the output image is displayed, the new pixels are evenly spaced, asindicated by s11 to s20 at the bottom of FIG. 29. Thus the leading andtrailing edge segments (b), which have a zoom ratio higher than n, areexpanded more than the interior edge segment (c), which has a zoom ratiolower than n. As a result, the expanded interior segment of the edge (c)appears sharper than it would appear if expanded by the conventionaluniform zoom ratio, while the leading and trailing segments of the edge(b) appear less sharp. Thus the sixth embodiment maintains edgesharpness by concentrating more of the edge variation into the interiorsegment of the expanded edge.

When an image is reduced, by increasing the zoom ratio in the leadingand trailing segments of an edge, the sixth embodiment mitigates theproblem of pixel dropout in the leading and trailing edge segments.

Vertical zooming is performed in the same way as horizontal zooming. Thehorizontal zoom ratio and vertical zoom ratio are mutually independent.The value of the parameter k may also differ between the horizontal andvertical directions.

The value of the parameter k can be increased to enhance edge sharpnessin the output image, or decreased to reduce edge sharpness. The sixthembodiment can be used to control edge sharpness in this way even whenthe basic zoom ratio (n) is equal to unity.

In the description above, the product of the first derivative, the thirdderivative, and the parameter k was simply added to the basic zoom ration, but more complex schemes are possible. For example, a positive upperlimit and a negative lower limit may be placed on the product, or anon-linear function may be applied as a substitute for multiplication byk.

Referring to FIG. 30, horizontal zooming and vertical zooming may alsobe carried out simultaneously in the sixth embodiment. The zoomprocessor 6 then comprises a two-dimensional interpolator 17, ahigh-frequency information detector 50, and a two-dimensional zoom ratiocontrol unit 51. The high-frequency information detector 50 detectshorizontal and vertical variations in the input image data andcalculates a two-dimensional first derivative d1 and third derivatived3. The two-dimensional zoom ratio control unit 51 calculates atwo-dimensional zoom ratio c4 from these derivatives d1 and d3. Thetwo-dimensional interpolator 17 uses the two-dimensional zoom ratio c4for two-dimensional (horizontal and vertical) interpolation. Furtherdetails will be given in the tenth embodiment.

FIG. 31 shows the internal configuration of the zoom processor 6 in theseventh embodiment of the invention. The vertical interpolator 11,vertical high-frequency information detector 12, horizontal interpolator14, and horizontal high-frequency information detector 15 are similar tothe corresponding elements in the first embodiment. The vertical zoomratio control unit 52 operates on the vertical first derivative vd1 andsecond derivative vd2 output by the vertical high-frequency informationdetector 12 to generate a vertical zoom ratio vc5. The horizontal zoomratio control unit 53 operates on the horizontal first derivative hd1and second derivative hd2 output by the vertical high-frequencyinformation detector 12 to generate a horizontal zoom ratio hc5.

The horizontal zoom ratio control unit 53 has the internal structureshown in FIG. 32, comprising a pair of absolute value calculators 54,55, a subtractor 56, a multiplier 57, and an adder 58. Absolute valuecalculator 54 takes the absolute value of the horizontal firstderivative hd1. Absolute value calculator 55 takes the absolute value ofthe horizontal second derivative hd2. The subtractor 56 subtracts theabsolute value of the horizontal first derivative from the absolutevalue of the second derivative. The multiplier 57 multiplies theresulting difference by a positive constant parameter k. The adder 58adds the resulting product to the basic zoom ratio n to obtain thehorizontal zoom ratio hc5.

The vertical zoom ratio control unit 52 has a similar internalstructure.

The operation of the seventh embodiment is illustrated in FIGS. 33A to33G. FIG. 33A shows an example of the brightness levels in a horizontalline in the vertically zoomed image data Pv. FIG. 33B shows thecorresponding horizontal first derivative hd1 and FIG. 33C shows thecorresponding horizontal second derivative hd2 as output by the verticalhigh-frequency information detector 12. FIG. 33D shows the absolutevalue of the horizontal first derivative abs(hd1) as calculated byabsolute value calculator 54. FIG. 33E shows the absolute value of thehorizontal second derivative abs(hd2) as calculated by absolute valuecalculator 55. FIG. 33F shows the horizontal zoom ratio hc5, which iscalculated from these absolute values by the following equation.

hc5=n+(k×((abs(hd2)−abs(hd1)))

As can be seen from FIG. 33F, the seventh embodiment has the samegeneral effect as the sixth embodiment, increasing the zoom ratio inleading and trailing edge segments (b) and reducing the zoom ratio ininterior edge segments (c), so that the average value of the zoom ratioover the entire horizontal line is equal to the basic zoom ratio (n)used in uniform segments (a).

AVE(hc5)=n

FIG. 33G depicts expanded output image data Po obtained by interpolationwith the variable zoom ratio hc5.

Vertical zooming is performed in the same way as horizontal zooming. Thehorizontal zoom ratio and vertical zoom ratio are mutually independent,and different values of the parameter k may be used in the horizontaland vertical directions.

The third derivatives employed in the sixth embodiment are calculated bydifferentiating the second derivatives employed in the seventhembodiment. Compared with the sixth embodiment, the seventh embodimentrequires one less differentiation operation in each direction, and canaccordingly be implemented more simply.

Referring to FIG. 34, horizontal zooming and vertical zooming may becarried out simultaneously. The zoom processor 6 then comprises atwo-dimensional interpolator 17, a high-frequency information detector59, and a two-dimensional zoom ratio control unit 60. The high-frequencyinformation detector 59 detects horizontal and vertical variations inthe input image data and calculates a two-dimensional first derivatived1 and second derivative d2. The two-dimensional zoom ratio control unit60 calculates a two-dimensional zoom ratio c5 from these derivatives d1,d2, taking differences between their absolute values as described above.The two-dimensional interpolator 17 uses the two-dimensional zoom ratioc5 for two-dimensional (horizontal and vertical) interpolation. Furtherinformation will be given in the description of the tenth embodiment.

FIG. 35 shows the internal configuration of the zoom processor 6 in aneighth embodiment of the invention. The vertical interpolator 11 andhorizontal interpolator 14 are similar to the corresponding elements inthe first embodiment.

The vertical high-frequency information detector 61 comprises a verticalhigh-pass spatial filter 62 and a vertical differentiator 63. Thevertical high-pass spatial filter 62 extracts high spatial frequenciesof the input image data Pi in the vertical direction and generates avertical high-frequency component of the image data, denoted Va(y). Thevertical differentiator 63 takes the first derivative of the input imagedata Pi in the vertical direction and generates a vertical firstderivative, denoted Vd(y). This Vd(y) may be identical to the firstderivative denoted vd1 in preceding embodiments, but the notation Vd(y)will be used for consistency with Va(y).

The vertical zoom rate control unit 64 receives the verticalhigh-frequency image component Va(y) and vertical first derivativeVd(y), and generates a vertical zoom ratio Vc(y). The verticalinterpolator 11 expands or reduces the input image Pi verticallyaccording to this zoom ratio Vc(y), to generate vertically zoomed imagedata Pv.

The horizontal high-frequency information detector 65 comprises ahorizontal high-pass spatial filter 66 and a horizontal differentiator67. The horizontal high-pass spatial filter 66 extracts high spatialfrequencies of the vertically zoomed image data Pv in the horizontaldirection, and generates a horizontal high-frequency component of theimage data, denoted Ha(x). The horizontal differentiator 67 takes thefirst derivative of vertically zoomed image data Pv in the horizontaldirection and generates a horizontal first derivative, denoted Hd(x).This Hd(x) may be identical to the first derivative denoted hd1 inpreceding embodiments, but the notation Hd(x) will be used forconsistency with Ha (x).

The horizontal zoom rate control unit 68 receives the horizontalhigh-frequency image component Ha(x) and horizontal first derivativeHd(x), and generates a horizontal zoom ratio Hc (x). The horizontalinterpolator 14 expands or reduces the vertically zoomed image data Pvhorizontally according to this zoom ratio Hc(x), to generate outputimage data Po.

FIGS. 36A to 36E illustrate the operation of the eighth embodiment. Morespecifically, they illustrate the horizontal zooming operation. Thehorizontal axis in these drawings indicates pixel position in ahorizontal scanning line in the vertically zoomed image data Pv.

FIG. 36A illustrates the input values p(x) in the horizontal scanningline. FIG. 36B illustrates the corresponding horizontal high-frequencyimage component Ha(x) extracted by the horizontal high-pass spatialfilter 66. FIG. 36C illustrates the horizontal first derivative Hd(x)obtained by the horizontal differentiator 67. FIG. 36D illustrates thehorizontal zoom ratio Hc(x) output by the horizontal zoom rate controlunit 68. The horizontal zoom rate control unit 68 calculates thehorizontal zoom ratio according to the following equation, in which n isthe basic zoom ratio and k is a positive constant parameter thatcontrols edge sharpness.

Hc(x)=n+(k×Ha(x)×Hd(x))

As FIG. 36D shows, the horizontal zoom ratio Hc(x) is equal to n insegments (a) in which the pixel value does not change, is greater than nin leading edge segments (b), and is less than n in trailing edgesegments (c). Furthermore, the average value of the zoom ratio Hc(x)over the entire horizontal line is equal to the basic zoom ratio n.

FIG. 36E illustrates expanded image data Po obtained by interpolationwith the variable zoom ratio Hc(x). The interpolation operationgenerates new uniform segments (a′) expanded by the basic zoom ratio n,new leading edge segments (b′) expanded by a ratio greater than n, andnew trailing edge segments (c′) expanded by a ratio less than n.

FIG. 37 illustrates the generation of output image data at an edgecomprising three pixels (p1, p2, p3) in the image data Pv, when thebasic zoom ratio (n) is three. New image data are generated at seveninterpolation points (q1 to q7) using the type of linear filter responsecharacteristic F(x) illustrated in FIG. 2. The equally spacedinterpolation points q′1 to q′7 at which new image data would begenerated in conventional image expansion are also shown. Thus thedistance from q′l to q′2, for example, is one-third the distance from p1to p2.

As explained in the first embodiment, the zoom ratio Hc(x) is calculatedat the equally spaced interpolation points (q′1 to q′7), the reciprocalvalues of the calculated zoom ratios are taken, and the reciprocalvalues are cumulatively summed to determine the interpolation points (q1to q7) at which the new pixels will be generated. The new pixelsgenerated at points q1 to q7 are denoted p_(o)′1 to p_(o)′7. The pixelvalues are calculated by interpolation from the input data p(x), usingthe filter response characteristics F(x). When displayed, these pixelsare evenly spaced, as indicated by p_(o) 1 to p_(o) 7 at the bottom ofFIG. 37. As a result, the leading edge segment (b) is expanded more thanthe trailing edge segment (c).

Vertical zooming in the eighth embodiment is carried out insubstantially the same way as horizontal zooming. The vertical zoomratio Vc(y) is calculated according to the following equation.

Vc(y)=n+(k×Va(y)×Vd(y)

As can be seen by comparing FIG. 37 with FIG. 11, the eighth embodimentprovides generally the same effect as the first embodiment, but to theextent that the high-frequency image data Ha(x) differs from the secondderivative hd2, the positions at which new pixels are generated in theeighth embodiment differ from the positions at which new pixels aregenerated in the first embodiment. For example, p_(o)′1, p_(o)′2, andp_(o)′3 in FIG. 37 are more closely spaced than q11, q12, and q13 inFIG. 11.

As in the first embodiment, the parameter k can be used to control edgesharpness in the output image even when the basic zoom ratio (n) isequal to unity. The values of n and k used in horizontal zooming may beindependent of the value of n and k used in vertical zooming.

In a variation of the eighth embodiment, the zoom processor 6 has thestructure shown in FIG. 38. The vertical high-frequency informationdetector 61 comprises the vertical differentiator 63 described above, avertical low-pass spatial filter 69, and a subtractor 70. The verticallow-pass spatial filter 69 generates vertical low-frequency image dataVs (y). The subtractor 70 takes the difference between the input imagedata Pi and the vertical low-frequency image data Vs(y), therebygenerating vertical high-frequency image data Va(y) for supply to thevertical zoom rate control unit 64. Similarly, the horizontalhigh-frequency information detector 65 comprises the horizontaldifferentiator 67 described above, a horizontal low-pass spatial filter71, and a subtractor 72. The horizontal low-pass spatial filter 71generates horizontal low-frequency image data Hs (x). The subtractor 72takes the difference between the vertically zoomed image data Pv and thehorizontal low-frequency image data Hs(x), thereby generating horizontalhigh-frequency image data Ha(x) for supply to the horizontal zoom ratecontrol unit 68. This variation generates the same output data Po as thestructure shown in FIG. 35.

FIG. 39 illustrates the structure of the zoom processor 6 in a ninthembodiment of the invention. The vertical interpolator 11, horizontalinterpolator 14, vertical zoom rate control unit 64, and horizontal zoomrate control unit 68 are similar to the corresponding elements in theeighth embodiment.

The vertical high-frequency information detector 73 in the ninthembodiment comprises a vertical differentiator 63 and a verticalbandpass spatial filter 73. The vertical bandpass spatial filter 73comprises a subtractor 70 and a pair of vertical low-pass spatialfilters 75, 76. The vertical low-pass spatial filters 75, 76 extract lowspatial frequencies in the vertical direction but operate with differentspatial cut-off frequencies, the cut-off frequency of the first verticallow-pass spatial filter 75 being lower than the cut-off frequency of thesecond vertical low-pass spatial filter 76. The subtractor 70 subtractsthe low-frequency image data Vs2(y) output by the second verticallow-pass spatial filter 76 from the low-frequency image data Vs1(y)output by the first vertical low-pass spatial filter 75, and suppliesthe resulting difference to the vertical zoom rate control unit 64. Thevertical zoom rate control unit 64 uses this difference in the same wayas the high-frequency image data Ha(y) received in the ninth embodiment.

Similarly, the horizontal high-frequency information detector 77comprises a horizontal differentiator 67 and a horizontal bandpassspatial filter 78. The horizontal bandpass spatial filter 78 comprises asubtractor 72 and a pair of horizontal low-pass spatial filters 79, 80.The horizontal low-pass spatial filters 79, 80 extract low spatialfrequencies in the horizontal direction but operate with differentcut-off frequencies, the cut-off frequency of the first horizontallow-pass spatial filter 79 being lower than the cut-off frequency of thesecond horizontal low-pass spatial filter 80. The subtractor 70subtracts the low-frequency image data Vs2(y) output by the secondhorizontal low-pass spatial filter 80 from the low-frequency image dataVs1(y) output by the first horizontal low-pass spatial filter 79, andsupplies the resulting difference to the horizontal zoom rate controlunit 68. The horizontal zoom rate control unit 68 uses this differencein the same way as the high-frequency image data Ha(x) received in theninth embodiment.

FIGS. 40A to 40G illustrate the operation of the ninth embodiment byillustrating the horizontal zooming operation. The horizontal axis inthese drawings indicates pixel position in a horizontal scanning line inthe vertically zoomed image data Pv.

FIG. 40A illustrates the input values p(x) in this horizontal scanningline. FIG. 40B illustrates the corresponding low-frequency data Hs1(x)output by the first horizontal low-pass spatial filter 79. FIG. 40Cillustrates the corresponding low-frequency data Hs2(x) output by thesecond horizontal low-pass spatial filter 80. FIG. 40D illustrates thedifference Hs1(x)−Hs2(x) obtained by the subtractor 72. FIG. 40Eillustrates the first derivative Hd(x) obtained by the horizontaldifferentiator 67. FIG. 40F illustrates the zoom ratio Hc(x) output bythe horizontal zoom rate control unit 68. The horizontal zoom ratecontrol unit 68 calculates the zoom ratio according to the followingequation, in which n is the basic zoom ratio and k is a constantparameter that controls edge sharpness.

Hc(x)=n+(k×(Hs1(x)−Hs2(x))×Hd(x))

As FIG. 40F shows, the zoom ratio Hc(x) has the same general form as inthe eighth embodiment, being equal to n in segments in which the pixelvalue does not change, greater than n in leading edge segments, and lessthan n in trailing edge segments, and having an average value equal to nover the entire horizontal scanning line.

FIG. 40G illustrates expanded image data Po obtained by interpolationwith the variable zoom ratio Hc(x). The result of interpolation isgenerally the same as in the eighth embodiment. However, the cut-offfrequencies of the first and second horizontal low-pass spatial filters79, 80 can be set to emphasize a particular range of spatial frequencycomponents at edges in the image.

Vertical zooming is carried out in the ninth embodiment in the same wayas horizontal zooming. The vertical zooming parameters (n, k, and thecut-off frequencies of the vertical low-pass spatial filters 75, 76) maydiffer from the horizontal zooming parameters (n, k, and the cut-offfrequencies of the horizontal low-pass spatial filters 79, 80).

FIG. 41 shows the structure of the zoom processor 6 in a tenthembodiment of the invention, in which horizontal and verticalinterpolation are performed simultaneously. This zoom processor 6comprises a two-dimensional interpolator 17 as described earlier, ahigh-frequency information detector 81 including a high-pass spatialfilter 82 and a two-dimensional differentiator 83, and a two-dimensionalzoom rate control unit 84.

The high-pass spatial filter 82 generates high-frequency image data a(x,y) by performing filtering computations at equally spaced interpolationpoints q′nm. These computations are carried out on input pixels that arehorizontally, vertically, and diagonally near to the interpolationpoints q′nm in the input image data Pi. The two-dimensionaldifferentiator 83 generates first-derivative data d(x, y) at the equallyspaced interpolation points q′nm, again using data (Pi) for input pixelsthat are horizontally, vertically, and diagonally nearby. Thetwo-dimensional zoom rate control unit 84 generates a zoom ratio c(x, y)for each of the equally spaced interpolation points q′nm from thehigh-frequency image data a(x, y) and first-derivative data d(x, y). Thezoom ratio c(x, y) is calculated according to the following equation.

c(x,y)=n+(k×a(x,y)×d(x,y))

The first derivative d(x, y) and zoom ratio c(x, y) are two-dimensionalquantities with horizontal and vertical components. The two-dimensionalinterpolator 17 uses the zoom ratios c(x, y) at the equally spacedinterpolation points q′nm to determine the interpolation points qnm atwhich new image data will be generated. FIG. 42 shows an example inwhich an equally spaced interpolation point q′nm is disposed at thecenter of a square formed by four pixels (p11, pl2, p21, p22) in theinput image data Pi, and the corresponding interpolation point qnmdetermined from the zoom ratios is disposed closer to pixel p11. Thetwo-dimensional interpolator 17 generates interpolation filteringcoefficients for this point qnm, or obtains the necessary coefficientsfrom a look-up table, and performs an interpolation calculation byapplying these coefficients to the input pixels (p11, pl2, p21, p22, inthis case) closest to interpolation point qnm.

The calculation is explained in FIG. 43. G(x) is the filter responsecharacteristic from which the interpolation coefficients are obtained.The distance between diagonally opposite pairs of input pixels isassumed to be unity. If the distances from the point qnm at which thenew image data will be generated to the four pixels p11, p12, p21, p22,are r₁₁, r₁₂, r₂₁, r₂₂, respectively, then the interpolated data valuepo′nm is calculated as follows.

po′nm=G(r ₁₁)p11+G(r ₁₂)p12+G(r ₂₁)p21+G(r ₂₂)p22

In the tenth embodiment, the accuracy of the interpolation calculationis enhanced because each interpolated value is obtained from the valuesof at least four of the pixels closest, in a two-dimensional sense, tothe interpolation point qnm.

In a variation of the tenth embodiment, shown in FIG. 44, thehigh-frequency information detector 81 comprises a two-dimensionaldifferentiator 83, a low-pass spatial filter 85, and a subtractor 86.The low-pass spatial filter 85 receives the input image data Pi andgenerates low-frequency image data s(x, y) at equally spacedinterpolation points. The subtractor 86 takes the difference between theinput image data Pi and the low-frequency image data s(x, y), therebygenerating the high-frequency image data a(x, y) that is supplied to thetwo-dimensional zoom rate control unit 84. At equally spacedinterpolation points not coinciding with input pixels, the subtractor 86takes the difference between s(x, y) and image data obtained from theinput image data Pi by, for example, linear interpolation.

In another variation of the tenth embodiment, shown in FIG. 45, thehigh-frequency information detector 87 comprises a two-dimensionaldifferentiator 83 and a bandpass spatial filter 88. The bandpass spatialfilter 88 comprises a subtractor 86 and a pair of low-pass spatialfilters 89, 90. The two low-pass spatial filters 89, 90 both extractlow-frequency data from the input image data Pi, but operate withdifferent cut-off frequencies, the cut-off frequency of the firstlow-pass spatial filter 89 being lower than the cut-off frequency of thesecond low-pass spatial filter 90. The subtractor 86 subtracts thelow-frequency image data s2(x, y) output by the second low-pass spatialfilter 90 from the low-frequency image data s1(x, y) output by the firstlow-pass spatial filter 89, and supplies the resulting difference to thetwo-dimensional zoom rate control unit 84. The vertical zoom ratecontrol unit 84 uses this difference as the two-dimensionalhigh-frequency image data a(x, y).

The invention can also be practiced when the input and output imagesignals are digital signals rather than analog signals. As an eleventhembodiment of the invention, FIG. 46 shows an image display apparatushaving an input terminal 91 for a digital image signal, that is, fordigital image data, and a digital display device 92. The apparatus alsohas an input terminal 2 for a synchronization signal, an imagepre-processor 4, a memory 5, and an image post-processor 7 as describedearlier, and a zoom processor 6 as described in any of the precedingembodiments. The image pre-processor 4 receives image data directly frominput terminal 91. If the image data are received in a coded form, theimage pre-processor 4 decodes the data. The digital display device 92receives image data directly from the image post-processor 7. A controlunit 93 receives the synchronization signal from input terminal 2, andcontrols the digital display device 92 and other elements by generatingthe necessary clock, timing, and control signals.

The operation of the eleventh embodiment can be understood from thedescription of the operation of the preceding embodiments, so furtherdetails will be omitted.

In a variation of the eleventh embodiment, an analog display device 9and a digital-to-analog converter 8 are used as in FIG. 6.

The invention can also be practiced in software, using a general-purposecomputer, for example, or a computing device embedded in an imagedisplay apparatus, instead of specialized arithmetic and logic circuits.As a twelfth embodiment, FIG. 47 shows a flowchart illustrating theoperation of a program that executes the functions of the zoom processor6 in any one of the first nine embodiments.

The program is divided into a first part (A) that performs verticalzooming, and a second part (B) that performs horizontal zooming. Bothparts scan the image in the usual scanning sequence, from left to rightand from top to bottom.

The first step (S1) in part A is to obtain the data needed forcalculating vertical high-frequency information at the current scanningpoint of the image. The necessary data include the values of two or morevertically adjacent pixels in the input image data (Pi), which are readfrom a memory such as the memory 5 in FIG. 46.

Vertical high-frequency information is calculated in the next step (S2),which detects pixel level variations in the vertical direction. Examplesof the types of high-frequency information that may be obtained includea first derivative (vd1), second derivative (vd2), third derivative(vd3), vertical variation pattern information (vp1), high-frequencyimage data (Va(y)), and a difference between two low-frequencycomponents (Vs1(y)−Vs2(y)), as described in the preceding embodiments.

A vertical zoom ratio (vc1) is then calculated from the verticalhigh-frequency information and the basic vertical zoom ratio (n), asdescribed in any of the first nine embodiments (step S3), and verticalinterpolation filtering calculations are performed according to thecalculated vertical zoom ratio to generate vertically zoomed image dataPv (step S4). The vertically zoomed image data Pv are stored temporarilyin, for example, the memory 5, or another memory not shown in thepreceding drawings.

The above process is repeated (step S5) until the end of the currentscanning line is reached; then the next scanning line is processed inthe same way (step S6). When the last scanning line has been processed,part A of the program ends and part B begins.

The first step (S7) in part B is to obtain the data needed forcalculating horizontal high-frequency information at the currentscanning point of the image. The necessary data include the values oftwo or more horizontally adjacent pixels in the image data (Pv)generated in part A.

Horizontal high-frequency information is calculated in the next step(S8), which detects pixel level variations in the horizontal direction.Examples of the types of high-frequency information that may be obtainedinclude a first derivative (hd1), second derivative (hd2), thirdderivative (hd3), horizontal variation pattern information (hp1),high-frequency image data (Ha(x)), and a difference between twolow-frequency components (Hs1(x)−Hs2(x)), as described in the precedingembodiments. A horizontal zoom ratio (hc1) is then calculated from thehorizontal high-frequency information and the basic horizontal zoomratio, as described in any of the preceding embodiments (step S9), andhorizontal interpolation filtering calculations are performed accordingto the calculated horizontal zoom ratio to generate new image data Pofor output to the image post-processor 7 (step S10). These steps arerepeated (step S11) until the end of the current scanning line isreached; then the next scanning line is processed in the same way untilthe last line has been processed (step S12) and the program ends.

In a variation of the twelfth embodiment, horizontal zooming isperformed before vertical zooming. That is, the order of parts A and Bof the program are interchanged. One part of the program, either part Aor part B, may also be omitted, if zooming is necessary in only onedirection.

In another variation of the twelfth embodiment, zooming is performedtwo-dimensionally, as in the tenth embodiment, for example, so that theimage has to be scanned and zoomed only once.

It is also possible to vary the scanning sequence by scanning the imagefrom right to left, or from bottom to top.

The basic zoom ratio (n) used in the vertical direction need not be thesame as the basic zoom ratio in used in the horizontal direction, butthe zoom ratios calculated in steps S3 and S9 should vary in such a waythat the correct total number of output pixels is obtained in eachvertical column and in each horizontal scanning line.

The computer or computing device that executes the program illustratedin FIG. 47 may also be programmed to carry out the functions of theimage pre-processor 4 and image post-processor 7 in FIG. 46.

In all of the preceding embodiments, the terms leading edge segment andtrailing edge segment merely denote two opposite parts of a portion ofthe image in which the pixel level varies. It is not necessary for theleading edge segment to be disposed to the left of the trailing edgesegment, or above it, or to precede it in the scanning sequence.

The preceding embodiments have been described as generating a variablezoom ratio with an average value equal to the basic zoom ratio (n), butthis condition can be modified to make the average spacing betweeninterpolation points equal to the basic spacing used in uniformsegments, or to make the average value of the reciprocal of the zoomratio equal to the reciprocal of the basic zoom ratio (1/n).

When a color image is processed, the operations described above may beperformed separately for each primary color, or separately for theluminance and chrominance components of the image signal.

Those skilled in the art will recognize that further variations arepossible within the scope claimed below.

What is claimed is:
 1. A method of processing an input image to obtainan output image, the input image being formed from input pixels havingbrightness levels, the method comprising the steps of: detectingpixel-to-pixel variations in the brightness levels in at least onedirection in the input image, thereby generating high spatial frequencyinformation; setting interpolation points with spacing varying accordingto the high spatial frequency information; generating output pixels fromthe input pixels by interpolation at the interpolation points; whereinsaid detecting includes calculating a first derivative of the brightnesslevels in said one direction; performing two low-pass spatial filteringoperations, with different cut-off frequencies, to obtain two lowspatial frequency components of the image; and taking a differencebetween said two low spatial frequency components.